Amazon EC2のベンチマークをCloudFormationで一発取得ができるようにしてみた
ウィスキー、シガー、パイプをこよなく愛する大栗です。
最近、EC2の新しいインスタンスタイプが出るたびにUnixBenchのスコアを見てニヤニヤすることが趣味になってきています。でも一々インスタンスを起動して、UnixBenchをインストールしてベンチマークするのが面倒になってきたのでCloudFormationで一発取得できるようにしてみました。
概要
以下のように動作します。
- CloudFormationテンプレートからStackを起動して、SpotインスタンスのAuto Scaling構成のEC2を起動します
- UnixBenchをインストールして、ベンチマークを実行します。
- UnixBenchの実行結果をS3へアップロードします。
- ベンチマーク後にCloudFormationのStackを削除します。
ポイントは、Auto Scalingにしてサンプル数を増やせること、Spotインスタンスを使用してコストを抑えることです。
テンプレート
CloudFormationのテンプレートは、以下のようになっています。 UnixBenchは16コア以上に対応していないため、テンプレート内でパッチを適用しています。
{ "AWSTemplateFormatVersion": "2010-09-09", "Description": "UnixBench Benchmarking template", "Parameters": { "ServiceName": { "Description": "Name for the service", "Type": "String", "Default" : "Benchmarking", "AllowedPattern": "[-_ a-zA-Z0-9]*", "ConstraintDescription": "can contain only alphanumeric characters, spaces, dashes and underscores." }, "VPC" : { "Description" : "VPC ID", "Type" : "AWS::EC2::VPC::Id" }, "TargetSubnet" : { "Type" : "AWS::EC2::Subnet::Id", "Description" : "Launch subnet Id" }, "TargetS3Bucket" : { "Type" : "String", "Description" : "The S3 Bucket to put the result", "MinLength" : "1" }, "TargetS3Prefix" : { "Type" : "String", "Default" : "benchmarking/", "Description" : "The Prefix result" }, "KeyName": { "Description": "Name of an existing EC2 KeyPair to enable SSH access to the instances", "Type": "AWS::EC2::KeyPair::KeyName" }, "InstanceCount" : { "Type" : "Number", "Description" : "Benchmark Instance Count", "MinValue" : "1", "MaxValue" : "10" }, "InstanceType" : { "Type" : "String", "Description" : "Benchmark Instance Type", "Default" : "t2.small" }, "AllocatedStorage" : { "Type" : "Number", "Description" : "Root volume size [GB]", "MinValue" : "8", "Default" : "1000" }, "SpotPrice" : { "Type" : "Number", "Description" : "spot price for instances", "MaxValue" : "10", "Default" : "1" } }, "Mappings": { "AWSAmazonLinuxAMI": { "us-east-1": { "name":"Virginia", "201409": "ami-ee6e2a86", "201503": "ami-1ecae776" }, "us-west-2": { "name":"Oregon", "201409": "ami-c1c39af1", "201503": "ami-e7527ed7" }, "us-west-1": { "name":"California", "201409": "ami-5a90891f", "201503": "ami-d114f295" }, "eu-west-1": { "name":"Ireland", "201409": "ami-8723aef0", "201503": "ami-a10897d6" }, "eu-central-1": { "name":"Frankfurt", "201409": "ami-0a003317", "201503": "ami-a8221fb5" }, "ap-southeast-1": { "name":"Singapole", "201409": "ami-94bb90c6", "201503": "ami-68d8e93a" }, "ap-northeast-1": { "name":"Tokyo", "201409": "ami-1e86981f", "201503": "ami-cbf90ecb" }, "ap-southeast-2": { "name":"Sydney", "201409": "ami-d70773ed", "201503": "ami-fd9cecc7" }, "sa-east-1": { "name":"SaoPaulo", "201409": "ami-a59925b8", "201503": "ami-b52890a8" } } }, "Conditions" : { "IsNokey" : { "Fn::Equals" : [ { "Ref" : "KeyName" }, "" ] } }, "Resources": { "BenchmarkEC2Role" : { "Type" : "AWS::IAM::Role", "Properties" : { "AssumeRolePolicyDocument" : { "Statement": [ { "Effect": "Allow", "Principal": { "Service": [ "ec2.amazonaws.com" ] }, "Action": [ "sts:AssumeRole" ] } ] }, "Path" : "/", "Policies" :[ { "PolicyName" : "BenchmarkEC2Policy", "PolicyDocument" : { "Statement": [ { "Sid": "AdminStmt", "Effect": "Allow", "Action": "*", "Resource": "*" } ] } }] } }, "BenchmarkInstanceIAMProfile" : { "Type" : "AWS::IAM::InstanceProfile", "Properties" : { "Path" : "/", "Roles" : [ { "Ref" : "BenchmarkEC2Role" } ] } }, "BenchmarkEC2SecurityGroup" : { "Type" : "AWS::EC2::SecurityGroup", "Properties" : { "VpcId" : { "Ref" : "VPC" }, "GroupDescription" : "Deny all communications in VPC", "SecurityGroupIngress" : [ ], "Tags" : [ { "Key" : "Name", "Value" : { "Fn::Join" : [ "-", [ { "Ref" : "ServiceName"}, "Bench" ] ] } } ] } }, "BenchmarkingAutoScalingGroup" : { "Type" : "AWS::AutoScaling::AutoScalingGroup", "Properties" : { "DesiredCapacity" : { "Ref" : "InstanceCount" }, "HealthCheckType" : "EC2", "LaunchConfigurationName" : { "Ref" : "BenchmarkingLaunchConfiguration"}, "MaxSize" : "10", "MinSize" : "1", "Tags" : [ {"Key" : "Name", "Value" : { "Fn::Join" : [ "-", [ { "Ref" : "ServiceName"}, "Benchmark" ] ] }, "PropagateAtLaunch" : true } ], "VPCZoneIdentifier" : [ { "Ref" : "TargetSubnet" } ] }, "DependsOn" : "BenchmarkingLaunchConfiguration" }, "BenchmarkingLaunchConfiguration" : { "Type" : "AWS::AutoScaling::LaunchConfiguration", "Properties" : { "InstanceType" : { "Ref" : "InstanceType" }, "BlockDeviceMappings" : [ { "DeviceName" : "/dev/xvda", "Ebs" : { "VolumeSize" : { "Ref" : "AllocatedStorage" } } }, { "DeviceName" : "/dev/xvdb", "VirtualName" : "ephemeral0" } ], "IamInstanceProfile" : { "Ref" : "BenchmarkInstanceIAMProfile" }, "KeyName" : { "Fn::If": [ "IsNokey", { "Ref" : "AWS::NoValue" }, { "Ref" : "KeyName" } ] }, "AssociatePublicIpAddress" : true, "SecurityGroups" : [ { "Ref" : "BenchmarkEC2SecurityGroup" } ], "SpotPrice" : { "Ref" : "SpotPrice" }, "ImageId" : { "Fn::FindInMap" : [ "AWSAmazonLinuxAMI", { "Ref" : "AWS::Region" }, "201503" ] }, "UserData" : { "Fn::Base64" : { "Fn::Join" : [ "", [ "#! /bin/bash -v\n", "yum update -y\n", "ln -sf /usr/share/zoneinfo/Asia/Tokyo /etc/localtime\n", "rpm -ivh http://repo.classmethod.info/yum/x86_64/cm-repo-release-0.1.0-1.noarch.rpm", "\n", "yum install -y gcc perl perl-Time-HiRes make tmux ec2-swap patch ", "\n", "cd /home/ec2-user", "\n", "wget https://byte-unixbench.googlecode.com/files/UnixBench5.1.3.tgz","\n", "tar xvzf UnixBench5.1.3.tgz", "\n", "cd /home/ec2-user/UnixBench", "\n", "# Patch for UnixBench ", "\n", "patch /home/ec2-user/UnixBench/Run << 'EOL'", "\n", "changeset: 3:565c69c33f98", "\n", "tag: tip", "\n", "user: Steven Noonan <[email protected]>", "\n", "date: Sun Mar 06 04:12:12 2011 -0800", "\n", "", "\n", "diff -r b50d02e5c913 -r 565c69c33f98 Run", "\n", "--- a/Run\tSun Mar 06 02:03:57 2011 -0800", "\n", "+++ b/Run\tSun Mar 06 04:12:12 2011 -0800", "\n", "@@ -105,10 +105,10 @@", "\n", " ", "\n", " # Configure the categories to which tests can belong.", "\n", " my $testCats = {", "\n", "- 'system' => { 'name' => \"System Benchmarks\", 'maxCopies' => 16 },", "\n", "+ 'system' => { 'name' => \"System Benchmarks\", 'maxCopies' => 0 },", "\n", " '2d' => { 'name' => \"2D Graphics Benchmarks\", 'maxCopies' => 1 },", "\n", " '3d' => { 'name' => \"3D Graphics Benchmarks\", 'maxCopies' => 1 },", "\n", "- 'misc' => { 'name' => \"Non-Index Benchmarks\", 'maxCopies' => 16 },", "\n", "+ 'misc' => { 'name' => \"Non-Index Benchmarks\", 'maxCopies' => 0 },", "\n", " };", "\n", " ", "\n", " ", "\n", "@@ -1328,7 +1328,7 @@", "\n", " # If the benchmark doesn't want to run with this many copies, skip it.", "\n", " my $cat = $params->{'cat'};", "\n", " my $maxCopies = $testCats->{$cat}{'maxCopies'};", "\n", "- next if ($copies > $maxCopies);", "\n", "+ next if ($maxCopies > 0 && $copies > $maxCopies);", "\n", " ", "\n", " # Run the benchmark.", "\n", " my $bresult = runBenchmark($bench, $params, $verbose, $logFile, $copies);", "\n", "EOL", "\n", "tmux -c '( sleep 300 ; ./Run ) > /home/ec2-user/benchmark_`curl -s http://169.254.169.254/latest/meta-data/instance-type`_", "`curl -s http://169.254.169.254/latest/meta-data/instance-id`_`date +%Y%m%d%H%M%S`.log' ","\n", "aws s3 cp /home/ec2-user/benchmark_*.log s3://", { "Ref" : "TargetS3Bucket"}, "/", { "Ref" : "TargetS3Prefix" }, "\n", "echo \"aws cloudformation delete-stack --region ", { "Ref" : "AWS::Region" }, " --stack-name ", { "Ref" : "AWS::StackName" }, " \" | at now + 5 minutes" ]]}} } } } }
ベンチマーク内容
ベンチマークは、以下の内容で実施しています。
- AMI:Amazon Linux AMI 2015.03.0 x86_64 HVM GP2
- インスタンスタイプ:可変(HVM対応のみ)
- ディスク:gp2が1ボリューム(サイズはパラメータで可変)
- UnixBench:UnixBench 5.1.3(パッチ適用版)
インスタンスタイプのHVM対応の可否はAmazon EC2再入門 2015年7月版をご確認下さい。
実行する
Select Template
このテンプレートを起動すると以下のようになります。
Specify Parameters
このStackのパラメータは以下のようになっています。
パラメータ | 内容 |
---|---|
AllocatedStorage | ルートボリュームのサイズです。ストレージタイプはgp2です。デフォルトは"1,000GB"です。 |
InstanceCount | ベンチマークを実施するインスタンス数です。デフォルトは"1"で、最大値は"10"です。 |
InstanceType | ベンチマーク対象のインスタンスタイプです。デフォルトは"c3.large"です。なお使用するAMIはHVMなのでHVM対応インスタンス以外は正常に起動しません。 |
KeyName | EC2に適用するKey Pairです。 |
SpotPrice | 適用するSpot価格です。デフォルトは"$1"です。 |
TargetS3Bucket | 結果をアップロードするS3バケットです。実行するアカウントのバケット名を入力してください。 |
TargetS3Prefix | 結果をアップロードする場所を指定します。デフォルトは"benchmarking/"です。 |
TargetSubnets | EC2を起動するサブネットを選択します。 |
VPC | EC2を起動するVPCを選択します。 |
Options
Optionsはそのままで進めます。
Review
Reviewでは、一番下のI acknowledge that this template might cause AWS CloudFormation to create IAM resources.
にチェックを入れます。
Stackが起動しました。ベンチマークが終了するまで1時間ほど待ちます。なお、実行後にベンチマークのStackは自動で消えるので安心してください。
ベンチマーク結果
S3の対象パスに実行結果のファイルが出力されます。
gcc -o ./pgms/arithoh -DTIME -Wall -pedantic -ansi -O2 -fomit-frame-pointer -fforce-addr -ffast-math -Wall -Darithoh ./src/arith.c gcc -o ./pgms/register -DTIME -Wall -pedantic -ansi -O2 -fomit-frame-pointer -fforce-addr -ffast-math -Wall -Ddatum='register int' ./src/arith.c gcc -o ./pgms/short -DTIME -Wall -pedantic -ansi -O2 -fomit-frame-pointer -fforce-addr -ffast-math -Wall -Ddatum=short ./src/arith.c gcc -o ./pgms/int -DTIME -Wall -pedantic -ansi -O2 -fomit-frame-pointer -fforce-addr -ffast-math -Wall -Ddatum=int ./src/arith.c gcc -o ./pgms/long -DTIME -Wall -pedantic -ansi -O2 -fomit-frame-pointer -fforce-addr -ffast-math -Wall -Ddatum=long ./src/arith.c gcc -o ./pgms/float -DTIME -Wall -pedantic -ansi -O2 -fomit-frame-pointer -fforce-addr -ffast-math -Wall -Ddatum=float ./src/arith.c gcc -o ./pgms/double -DTIME -Wall -pedantic -ansi -O2 -fomit-frame-pointer -fforce-addr -ffast-math -Wall -Ddatum=double ./src/arith.c gcc -o ./pgms/hanoi -DTIME -Wall -pedantic -ansi -O2 -fomit-frame-pointer -fforce-addr -ffast-math -Wall ./src/hanoi.c gcc -o ./pgms/syscall -DTIME -Wall -pedantic -ansi -O2 -fomit-frame-pointer -fforce-addr -ffast-math -Wall ./src/syscall.c gcc -o ./pgms/context1 -DTIME -Wall -pedantic -ansi -O2 -fomit-frame-pointer -fforce-addr -ffast-math -Wall ./src/context1.c gcc -o ./pgms/pipe -DTIME -Wall -pedantic -ansi -O2 -fomit-frame-pointer -fforce-addr -ffast-math -Wall ./src/pipe.c gcc -o ./pgms/spawn -DTIME -Wall -pedantic -ansi -O2 -fomit-frame-pointer -fforce-addr -ffast-math -Wall ./src/spawn.c gcc -o ./pgms/execl -DTIME -Wall -pedantic -ansi -O2 -fomit-frame-pointer -fforce-addr -ffast-math -Wall ./src/execl.c cd ./src; gcc -c -DTIME -Wall -pedantic -ansi -DHZ= -O2 -fomit-frame-pointer -fforce-addr -ffast-math -Wall dhry_1.c cd ./src; gcc -c -DTIME -Wall -pedantic -ansi -DHZ= -O2 -fomit-frame-pointer -fforce-addr -ffast-math -Wall dhry_2.c gcc -o ./pgms/dhry2 -DTIME -Wall -pedantic -ansi -O2 -fomit-frame-pointer -fforce-addr -ffast-math -Wall ./src/dhry_1.o ./src/dhry_2.o cd ./src; rm -f dhry_1.o dhry_2.o cd ./src; gcc -c -DTIME -Wall -pedantic -ansi -DREG=register -DHZ= -O2 -fomit-frame-pointer -fforce-addr -ffast-math -Wall dhry_1.c -o dhry_1_reg.o cd ./src; gcc -c -DTIME -Wall -pedantic -ansi -DREG=register -DHZ= -O2 -fomit-frame-pointer -fforce-addr -ffast-math -Wall dhry_2.c -o dhry_2_reg.o gcc -o ./pgms/dhry2reg -DTIME -Wall -pedantic -ansi -O2 -fomit-frame-pointer -fforce-addr -ffast-math -Wall ./src/dhry_1_reg.o ./src/dhry_2_reg.o cd ./src; rm -f dhry_1_reg.o dhry_2_reg.o gcc -o ./pgms/looper -DTIME -Wall -pedantic -ansi -O2 -fomit-frame-pointer -fforce-addr -ffast-math -Wall ./src/looper.c gcc -o ./pgms/fstime -DTIME -Wall -pedantic -ansi -O2 -fomit-frame-pointer -fforce-addr -ffast-math -Wall ./src/fstime.c gcc -o ./pgms/whetstone-double -DTIME -Wall -pedantic -ansi -O2 -fomit-frame-pointer -fforce-addr -ffast-math -Wall -DDP -DUNIX -DUNIXBENCH ./src/whets.c -lm make all make[1]: Entering directory `/home/ec2-user/UnixBench' Checking distribution of files ./pgms exists ./src exists ./testdir exists ./results exists make[1]: Leaving directory `/home/ec2-user/UnixBench' # # # # # # # ##### ###### # # #### # # # # ## # # # # # # # ## # # # # # # # # # # # ## ##### ##### # # # # ###### # # # # # # ## # # # # # # # # # # # # ## # # # # # # # ## # # # # #### # # # # # ##### ###### # # #### # # Version 5.1.3 Based on the Byte Magazine Unix Benchmark Multi-CPU version Version 5 revisions by Ian Smith, Sunnyvale, CA, USA January 13, 2011 johantheghost at yahoo period com 1 x Dhrystone 2 using register variables 1 2 3 4 5 6 7 8 9 10 1 x Double-Precision Whetstone 1 2 3 4 5 6 7 8 9 10 1 x Execl Throughput 1 2 3 1 x File Copy 1024 bufsize 2000 maxblocks 1 2 3 1 x File Copy 256 bufsize 500 maxblocks 1 2 3 1 x File Copy 4096 bufsize 8000 maxblocks 1 2 3 1 x Pipe Throughput 1 2 3 4 5 6 7 8 9 10 1 x Pipe-based Context Switching 1 2 3 4 5 6 7 8 9 10 1 x Process Creation 1 2 3 1 x System Call Overhead 1 2 3 4 5 6 7 8 9 10 1 x Shell Scripts (1 concurrent) 1 2 3 1 x Shell Scripts (8 concurrent) 1 2 3 2 x Dhrystone 2 using register variables 1 2 3 4 5 6 7 8 9 10 2 x Double-Precision Whetstone 1 2 3 4 5 6 7 8 9 10 2 x Execl Throughput 1 2 3 2 x File Copy 1024 bufsize 2000 maxblocks 1 2 3 2 x File Copy 256 bufsize 500 maxblocks 1 2 3 2 x File Copy 4096 bufsize 8000 maxblocks 1 2 3 2 x Pipe Throughput 1 2 3 4 5 6 7 8 9 10 2 x Pipe-based Context Switching 1 2 3 4 5 6 7 8 9 10 2 x Process Creation 1 2 3 2 x System Call Overhead 1 2 3 4 5 6 7 8 9 10 2 x Shell Scripts (1 concurrent) 1 2 3 2 x Shell Scripts (8 concurrent) 1 2 3 ======================================================================== BYTE UNIX Benchmarks (Version 5.1.3) System: ip-172-31-31-190: GNU/Linux OS: GNU/Linux -- 3.14.35-28.38.amzn1.x86_64 -- #1 SMP Wed Mar 11 22:50:37 UTC 2015 Machine: x86_64 (x86_64) Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8") CPU 0: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz (5600.2 bogomips) Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET CPU 1: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz (5600.2 bogomips) Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET 08:02:19 up 7 min, 0 users, load average: 0.01, 0.18, 0.13; runlevel 3 ------------------------------------------------------------------------ Benchmark Run: Thu Jul 30 2015 08:02:19 - 08:30:23 2 CPUs in system; running 1 parallel copy of tests Dhrystone 2 using register variables 35539934.8 lps (10.0 s, 7 samples) Double-Precision Whetstone 4421.1 MWIPS (9.9 s, 7 samples) Execl Throughput 3941.0 lps (30.0 s, 2 samples) File Copy 1024 bufsize 2000 maxblocks 986463.8 KBps (30.0 s, 2 samples) File Copy 256 bufsize 500 maxblocks 261828.2 KBps (30.0 s, 2 samples) File Copy 4096 bufsize 8000 maxblocks 2834228.7 KBps (30.0 s, 2 samples) Pipe Throughput 1743736.3 lps (10.0 s, 7 samples) Pipe-based Context Switching 33003.2 lps (10.0 s, 7 samples) Process Creation 11069.6 lps (30.0 s, 2 samples) Shell Scripts (1 concurrent) 7300.3 lpm (60.0 s, 2 samples) Shell Scripts (8 concurrent) 1204.2 lpm (60.0 s, 2 samples) System Call Overhead 2416342.8 lps (10.0 s, 7 samples) System Benchmarks Index Values BASELINE RESULT INDEX Dhrystone 2 using register variables 116700.0 35539934.8 3045.4 Double-Precision Whetstone 55.0 4421.1 803.8 Execl Throughput 43.0 3941.0 916.5 File Copy 1024 bufsize 2000 maxblocks 3960.0 986463.8 2491.1 File Copy 256 bufsize 500 maxblocks 1655.0 261828.2 1582.0 File Copy 4096 bufsize 8000 maxblocks 5800.0 2834228.7 4886.6 Pipe Throughput 12440.0 1743736.3 1401.7 Pipe-based Context Switching 4000.0 33003.2 82.5 Process Creation 126.0 11069.6 878.5 Shell Scripts (1 concurrent) 42.4 7300.3 1721.8 Shell Scripts (8 concurrent) 6.0 1204.2 2007.1 System Call Overhead 15000.0 2416342.8 1610.9 ======== System Benchmarks Index Score 1305.2 ------------------------------------------------------------------------ Benchmark Run: Thu Jul 30 2015 08:30:23 - 08:58:31 2 CPUs in system; running 2 parallel copies of tests Dhrystone 2 using register variables 40169627.7 lps (10.0 s, 7 samples) Double-Precision Whetstone 7356.5 MWIPS (9.9 s, 7 samples) Execl Throughput 6170.4 lps (30.0 s, 2 samples) File Copy 1024 bufsize 2000 maxblocks 1087235.9 KBps (30.0 s, 2 samples) File Copy 256 bufsize 500 maxblocks 283464.9 KBps (30.0 s, 2 samples) File Copy 4096 bufsize 8000 maxblocks 3409515.2 KBps (30.0 s, 2 samples) Pipe Throughput 2080164.4 lps (10.0 s, 7 samples) Pipe-based Context Switching 363967.5 lps (10.0 s, 7 samples) Process Creation 20402.8 lps (30.0 s, 2 samples) Shell Scripts (1 concurrent) 8923.3 lpm (60.0 s, 2 samples) Shell Scripts (8 concurrent) 1218.0 lpm (60.0 s, 2 samples) System Call Overhead 3031262.1 lps (10.0 s, 7 samples) System Benchmarks Index Values BASELINE RESULT INDEX Dhrystone 2 using register variables 116700.0 40169627.7 3442.1 Double-Precision Whetstone 55.0 7356.5 1337.5 Execl Throughput 43.0 6170.4 1435.0 File Copy 1024 bufsize 2000 maxblocks 3960.0 1087235.9 2745.5 File Copy 256 bufsize 500 maxblocks 1655.0 283464.9 1712.8 File Copy 4096 bufsize 8000 maxblocks 5800.0 3409515.2 5878.5 Pipe Throughput 12440.0 2080164.4 1672.2 Pipe-based Context Switching 4000.0 363967.5 909.9 Process Creation 126.0 20402.8 1619.3 Shell Scripts (1 concurrent) 42.4 8923.3 2104.6 Shell Scripts (8 concurrent) 6.0 1218.0 2030.0 System Call Overhead 15000.0 3031262.1 2020.8 ======== System Benchmarks Index Score 1991.2
ベンチマークが取りたい方は以下のリンクをクリックしましょう!
さいごに
これで簡単にベンチマークができるようになりました。しかしUnixBenchは古いベンチマークツールなのでCPUのコアが多数の場合は効率的に動作せずにスコアが伸びないようです。また細かな計測環境の違いによってもベンチマークのコスアが大きく変動することもあります。あくまでも参考情報であることを認識してベンチマークを行いましょう。